Control device and control method for mobile object, storage medium, and vehicle

ABSTRACT

A control device for a mobile object including one or more imaging devices, the control device includes an image acquisition unit configured to acquire an image of an external environment of the mobile object from the one or more imaging devices, a correction unit configured to perform distortion reduction processing for reducing distortion of an image for each of one or more regions included in an image acquired from the one or more imaging devices, and a recognition unit configured to recognize the external environment of the mobile object based on an image on which the distortion reduction processing has been performed. The correction unit is configured to determine the one or more regions to be a target of the distortion reduction processing in accordance with a predetermined rule according to a moving scene of the mobile object.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to and the benefit of Japanese PatentApplication No. 2021-049122 filed on Mar. 23, 2021, the entiredisclosure of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

The present disclosure relates to a control device and a control methodfor a mobile object, a storage medium, and a vehicle.

Description of the Related Art

A technique for recognizing an external environment of a vehicle using aplurality of cameras has been put into practical use. Recognitionresults of the external environment are used for driving assistance andautomated driving. Japanese Patent Laid-Open No. 2018-171964 proposes atechnique for capturing an image of surroundings of a vehicle in a widerange by a wide-angle lens camera. Coordinate transformation isperformed on an image captured by a wide-angle lens camera in order toreduce distortion.

SUMMARY OF THE INVENTION

Processing for reducing distortion of an image captured by a camera towhich a wide-angle lens or a fisheye lens is attached consumes power.Thus, if the processing of reducing the distortion is excessivelyexecuted in order to recognize the external environment of the vehicle,the power consumption increases. Such an increase in power consumptionis not limited to the vehicle, and is also applicable to other mobileobjects. Some aspects of the present disclosure provide a technology forappropriately recognizing an external environment of a mobile objectaccording to a moving scene.

According to an embodiment, a control device for a mobile objectincluding one or more imaging devices includes an image acquisition unitconfigured to acquire an image of an external environment of the mobileobject from the one or more imaging devices, a correction unitconfigured to perform distortion reduction processing for reducingdistortion of an image for each of one or more regions included in animage acquired from the one or more imaging devices, and a recognitionunit configured to recognize the external environment of the mobileobject based on an image on which the distortion reduction processinghas been performed. The correction unit is configured to determine theone or more regions to be a target of the distortion reductionprocessing in accordance with a predetermined rule according to a movingscene of the mobile object.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of avehicle according to an embodiment;

FIGS. 2A to 2D are schematic diagrams illustrating fields of view ofcameras according to the embodiment;

FIG. 3 is a schematic diagram illustrating distortion reductionprocessing according to the embodiment;

FIGS. 4A and 4B are schematic diagrams illustrating target regions ofthe distortion reduction processing according to the embodiment;

FIG. 5 is a flowchart illustrating an operation example of a controldevice for the vehicle according to the embodiment;

FIG. 6 is a timing chart illustrating target candidates for thedistortion reduction processing according to the embodiment;

FIG. 7 is a schematic diagram illustrating a normal traveling sceneaccording to the embodiment;

FIG. 8 is a schematic diagram illustrating a state transition inaccordance with a specific rule according to the embodiment;

FIGS. 9A to 9C are schematic diagrams illustrating a position in avertical direction of a region in accordance with a specific ruleaccording to the embodiment;

FIG. 10 is a schematic diagram illustrating a specific traveling sceneaccording to the embodiment;

FIG. 11 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 12 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIGS. 13A to 13D are schematic diagrams illustrating a specifictraveling scene according to the embodiment;

FIG. 14 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 15 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIGS. 16A to 16C are schematic diagrams illustrating a position in avertical direction of a region in accordance with a specific ruleaccording to the embodiment;

FIG. 17 is a schematic diagram illustrating a specific traveling sceneaccording to the embodiment;

FIG. 18 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 19 is a schematic diagram illustrating a specific traveling sceneaccording to the embodiment;

FIG. 20 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 21 is a schematic diagram illustrating a specific traveling sceneaccording to the embodiment;

FIG. 22 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 23 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment;

FIG. 24 is a schematic diagram illustrating a specific traveling sceneaccording to the embodiment; and

FIG. 25 is a schematic diagram illustrating state transition inaccordance with a specific rule according to the embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference tothe attached drawings. Note, the following embodiments are not intendedto limit the scope of the claimed invention, and limitation is not madeto an invention that requires a combination of all features described inthe embodiments. Two or more of the multiple features described in theembodiments may be combined as appropriate. Furthermore, the samereference numerals are given to the same or similar configurations, andredundant description thereof is omitted.

Note that in the following embodiment, description will be made assumingthat a mobile object is a vehicle, but the mobile object is not limitedto a vehicle, and may be a flying object, a robot, or the like.

FIG. 1 is a block diagram of a vehicle 1 according to an embodiment ofthe present invention. In FIG. 1, an outline of a vehicle 1 isillustrated in a plan view and a side view. The vehicle 1 is, forexample, a four-wheeled passenger vehicle of a sedan type. The vehicle 1may be such a four-wheeled vehicle, a two-wheeled vehicle, or anothertype of vehicle.

The vehicle 1 includes a vehicle control device 2 (hereinafter, simplyreferred to as a control device 2) that controls the vehicle 1. Thecontrol device 2 includes a plurality of electronic control units (ECUs)20 to 29 communicably connected by an in-vehicle network. Each ECUincludes a processor such as a central processing unit (CPU), a memorysuch as a semiconductor memory, an interface with an external device,and the like. The memory stores programs executed by the processor, dataused for processing by the processor, and the like. Each ECU may includea plurality of processors, memories, interfaces, and the like. Forexample, the ECU 20 includes a processor 20 a and a memory 20 b.Processing by the ECU 20 is executed by the processor 20 a executing acommand including the program stored in the memory 20 b. Alternatively,the ECU 20 may include a dedicated integrated circuit such as anapplication specific integrated circuit (ASIC) for executing processingby the ECU 20. A similar configuration applies to other ECUs.

Hereinafter, functions and the like assigned to each of the ECUs 20 to29 will be described. Note that the number of ECUs and the functionsassigned to the ECUs can be designed as appropriate and can besubdivided or integrated as compared with the present embodiment.

The ECU 20 executes control related to automated traveling of thevehicle 1. In automated driving, at least one of the steering andacceleration/deceleration of the vehicle 1 is automatically controlled.The automated traveling by the ECU 20 may include automated travelingthat does not require a traveling operation by a driver (which may alsobe referred to as automated driving) and automated traveling forassisting the traveling operation by the driver (which may also bereferred to as driving assistance).

The ECU 21 controls an electric power steering device 3. The electricpower steering device 3 includes a mechanism that steers a front wheelaccording to a driver's driving operation (steering operation) on asteering wheel 31. In addition, the electric power steering device 3includes a motor that exerts a driving force for assisting the steeringoperation and automatically steering the front wheel, a sensor thatdetects a steering angle, and the like. In a case where the drivingstate of the vehicle 1 is automated driving, the ECU 21 controls theelectric power steering device 3 in an automated manner in response toan instruction from the ECU 20, and controls the traveling direction ofthe vehicle 1.

The ECUs 22 and 23 perform control of a detection unit that detects asituation around the vehicle and information processing of a detectionresult. The vehicle 1 includes one standard camera 40 and four fisheyecameras 41 to 44 as the detection unit that detects a situation of thevehicle. The standard camera 40 and the fisheye cameras 42 and 44 areconnected to the ECU 22. The fisheye cameras 41 and 43 are connected tothe ECU 23. The ECUs 22 and 23 can extract the contour of a targetobject and a vehicle lane line (such as a white line) on the road byanalyzing images captured by the standard camera 40 and the fisheyecameras 41 to 44.

The fisheye cameras 41 to 44 are cameras to which a fisheye lens isattached. Hereinafter, a configuration of the fisheye camera 41 will bedescribed. Other fisheye cameras 42 to 44 may have similarconfigurations. An angle of view of the fisheye camera 41 is wider thanan angle of view of the standard camera 40. Thus, the fisheye camera 41can capture a wider range than the standard camera 40. The imagecaptured by the fisheye camera 41 has a large distortion compared to theimage captured by the standard camera 40. Thus, before analyzing theimage captured by the fisheye camera 41, the ECU 23 may performconversion processing (hereinafter, referred to as “distortion reductionprocessing”) for reducing distortion on the image. On the other hand,the ECU 22 does not need to perform the distortion reduction processingon the image captured by the standard camera 40 before analyzing theimage. Thus, the standard camera 40 is an imaging device for capturingan image not to be a target of the distortion reduction processing, andthe fisheye camera 41 is an imaging device for capturing an image to bea target of the distortion reduction processing. Instead of the standardcamera 40, another imaging device that captures an image not to be atarget of the distortion reduction processing, for example, a camera towhich a wide-angle lens or a telephoto lens is attached, may be used.

The standard camera 40 is attached at the center of a front portion ofthe vehicle 1 and captures an image of a situation ahead of the vehicle1. The fisheye camera 41 is attached at the center of the front portionof the vehicle 1 and captures an image of a situation ahead of thevehicle 1. In FIG. 1, the standard camera 40 and the fisheye camera 41are illustrated as being arranged in a horizontal direction. However,the arrangement of the standard camera 40 and the fisheye camera 41 isnot limited thereto, and may be, for example, arranged in a verticaldirection. In addition, at least one of the standard camera 40 and thefisheye camera 41 may be attached to a front portion of the roof (forexample, on the vehicle interior side of the windshield) of the vehicle1. For example, the fisheye camera 41 may be attached at the center ofthe front portion (for example, a bumper) of the vehicle 1, and thestandard camera 40 may be attached at the front portion of the roof ofthe vehicle 1. The fisheye camera 42 is attached at the center of aright side portion of the vehicle 1 and captures an image of a situationon the right of the vehicle 1. The fisheye camera 43 is attached at thecenter of a rear portion of the vehicle 1 and captures an image of asituation behind the vehicle 1. The fisheye camera 44 is attached at thecenter of a left side portion of the vehicle 1 and captures an image ofa situation on the left of the vehicle 1.

The type, number, and attachment position of the camera included in thevehicle 1 are not limited to the above-described examples. In addition,the vehicle 1 may include a light detection and ranging (LIDAR) or amillimeter wave radar as a detection unit for detecting a target objectaround the vehicle 1 and measuring the distance to the target object.

The ECU 22 controls the standard camera 40 and the fisheye cameras 42and 44 and performs information processing on detection results. The ECU23 controls the fisheye cameras 41 and 43 and performs informationprocessing on detection results. The reliability of the detectionresults can be improved by dividing the detection units for detectingthe situation of the vehicle into two systems.

The ECU 24 controls a gyro sensor 5, a global positioning system (GPS)sensor 24 b, and a communication device 24 c, and performs informationprocessing on detection results or communication results. The gyrosensor 5 detects a rotational motion of the vehicle 1. The detectionresult of the gyro sensor 5, the wheel speed, and the like enabledetermination of the course of the vehicle 1. The GPS sensor 24 bdetects the current location of the vehicle 1. The communication device24 c performs wireless communication with a server that provides mapinformation and traffic information and acquires these pieces ofinformation. The ECU 24 can access a database 24 a on map informationconstructed in the memory, and the ECU 24 searches for a route from thecurrent position to a destination and the like. The ECU 24, the mapdatabase 24 a, and the GPS sensor 24 b constitute a so-called navigationdevice.

The ECU 25 is provided with a communication device 25 a forinter-vehicle communication. The communication device 25 a performswireless communication with other surrounding vehicles to exchangeinformation between the vehicles.

The ECU 26 controls a power plant 6. The power plant 6 is a mechanismthat outputs a driving force for rotating driving wheels of the vehicle1 and includes, for example, an engine and a transmission. For example,the ECU 26 controls the output of the engine according to the drivingoperation (accelerator operation or acceleration operation) of thedriver detected by an operation detection sensor 7 a provided on anaccelerator pedal 7A and switches the gear ratio of the transmissionbased on information such as the vehicle speed detected by a vehiclespeed sensor 7 c and the like. When the driving state of the vehicle 1is automated driving, the ECU 26 automatically controls the power plant6 in response to an instruction from the ECU 20 and controls theacceleration and deceleration of the vehicle 1.

The ECU 27 controls lighting devices (headlights, taillights, and thelike) including direction indicators 8 (blinkers). In the example ofFIG. 1, the direction indicators 8 are provided at a front portion, doormirrors, and a rear portion of the vehicle 1.

The ECU 28 controls an input/output device 9. The input/output device 9outputs information to the driver and accepts an input of informationfrom the driver. A voice output device 91 notifies the driver ofinformation by voice. A display device 92 notifies the driver ofinformation by displaying an image. The display device 92 is arrangedon, for example, a front surface of a driver's seat, and constitutes aninstrument panel or the like. Note that, although the sound and thedisplay have been given as examples here, information may be notified byvibration or light. In addition, notification of information may beprovided by using a combination of some of the sound, the display, thevibration, and the light. Further, the combination or the notificationmode may vary in accordance with the level (for example, the degree ofurgency) of information that should be notified. An input device 93 isarranged at a position where the driver is able to operate, andconstitutes a switch group for giving an instruction to the vehicle 1,but may also include a voice input device.

The ECU 29 controls a brake device 10 and a parking brake (notillustrated). The brake device 10 is, for example, a disc brake device,is provided on each wheel of the vehicle 1, and applies resistance tothe rotation of the wheel to decelerate or stop the vehicle 1. The ECU29 controls working of the brake device 10 in response to the driver'sdriving operation (brake operation) that has been detected by anoperation detection sensor 7 b provided on a brake pedal 7B, forexample. When the driving state of the vehicle 1 is automated driving,the ECU 29 automatically controls the brake device 10 in response to aninstruction from the ECU 20 and controls the deceleration and stop ofthe vehicle 1. The brake device 10 and the parking brake are alsocapable of working to maintain a stopped state of the vehicle 1. Inaddition, when the transmission of the power plant 6 includes a parkinglock mechanism, it can also be operated to maintain the stopped state ofthe vehicle 1.

The fields of view of the standard camera 40 and the fisheye cameras 41to 44 will be described with reference to FIGS. 2A to 2D. FIG. 2Aillustrates fields of view in the horizontal direction of the respectivecameras, FIG. 2B illustrates a field of view in the vertical directionof the fisheye camera 41 in the front portion of the vehicle 1, FIG. 2Cillustrates a field of view in the vertical direction of the fisheyecamera 42 in the right side portion of the vehicle 1, and FIG. 2Dillustrates a field of view in the vertical direction of the fisheyecamera 43 in the rear portion of the vehicle 1. In the presentspecification, the horizontal direction and the vertical direction arebased on the vehicle body of the vehicle 1. The field of view in thevertical direction of the fisheye camera 42 on the left side portion ofthe vehicle 1 may be similar to that in FIG. 2C, and thus thedescription thereof is omitted.

First, fields of view of the vehicle 1 in a plan view (that is, thehorizontal direction of the vehicle 1) will be described with referenceto FIG. 2A. The standard camera 40 captures an image of a scene includedin a field of view 200. An image-capture center 200C of the standardcamera 40 faces a direction directly ahead of the vehicle 1. Theimage-capture center 200C may be defined by the direction of the opticalaxis of the lens. The angle of view in the horizontal direction of thestandard camera 40 may be less than 90°, and may be, for example, about45° or about 30°.

The fisheye camera 41 captures an image of a scene included in a fieldof view 201. The image-capture center 201C of the fisheye camera 41faces the direction directly ahead of the vehicle 1. The fisheye camera42 captures an image of a scene included in a field of view 202. Theimage-capture center 202C of the fisheye camera 42 faces a directiondirectly right of the vehicle 1. The fisheye camera 43 captures an imageof a scene included in a field of view 203. The image-capture center203C of the fisheye camera 43 faces a direction directly behind of thevehicle 1. The fisheye camera 44 captures an image of a scene includedin a field of view 204. The image-capture center 204C of the fisheyecamera 44 faces a direction directly left of the vehicle 1. The anglesof view in the horizontal direction of the fisheye cameras 41 to 44 maybe, for example, greater than 90°, greater than 150°, greater than 180°,or about 180°, for example. FIG. 2A illustrates an example in which theangle of view in the horizontal direction of the fisheye cameras 41 to44 is 180°.

Next, fields of view in the vertical direction of view of the vehicle 1will be described with reference to FIGS. 2B to 2D. FIG. 2B illustratesthe field of view in the vertical direction of the fisheye camera 41,FIG. 2C illustrates the vertical field of view of the fisheye camera 42,and FIG. 2D illustrates the vertical field of view of the fisheye camera43. The field of view in the vertical direction of the other fisheyecamera 44 may be similar to that of FIG. 2C.

The angles of view in the vertical direction of the fisheye cameras 41to 44 may be, for example, greater than 90°, greater than 150°, greaterthan 180°, or about 180°, for example. FIGS. 2B to 2D illustrate anexample in which the angle of view in the vertical direction of thefisheye cameras 41 to 44 is 180°. The image-capture center 203C of thefisheye camera 43 is directed downward (toward the ground) with respectto a direction parallel to the ground. Alternatively, the image-capturecenter 203C of the fisheye camera 43 may be directed in a directionparallel to the ground, or may be directed upward (opposite to theground) with respect to the direction parallel to the ground. Inaddition, the image-capture centers 201C to 204C of the fisheye cameras41 to 44 may be directed in different directions in the verticaldirection.

Since the standard camera 40 and the fisheye cameras 41 to 44 have thefields of view 200 to 204 as described above, each of the directiondirectly ahead of the vehicle 1 and four diagonal directions of thevehicle 1 is included in the fields of view of two separate cameras.Specifically, the direction directly ahead of the vehicle 1 is includedin both the field of view 200 of the standard camera 40 and the field ofview 201 of the fisheye camera 41. The direction right-diagonally aheadof the vehicle 1 is included in both the field of view 201 of thefisheye camera 41 and the field of view 202 of the fisheye camera 42.The same applies to the other three diagonal directions of the vehicle1.

The distortion reduction processing of images captured by the fisheyecameras 41 to 44 will be described with reference to FIGS. 3 to 4B. FIG.3 illustrates images before and after the distortion reductionprocessing. FIGS. 4A and 4B illustrate regions to be targets of thedistortion reduction processing. FIG. 4A is a top view of vehicle 1, andFIG. 4B is a rear view of vehicle 1. An image 300 is an image of asituation on the right of the vehicle 1 captured by the fisheye camera42. As illustrated in FIG. 3, the image 300 has a large distortionparticularly in peripheral portions.

The ECU 22 connected to the fisheye camera 42 performs the distortionreduction processing on the image 300. Specifically, as illustrated inFIG. 3, the ECU 22 determines one point in the image 300 as a transformcenter 301. As illustrated in FIGS. 4A and 4B, the transform center 301is located on the right side in the field of view 202 as viewed from thefisheye camera 42 in the horizontal direction, and is directed in adirection parallel to the ground in the vertical direction.

The ECU 22 cuts out a rectangular region 302 centered on the transformcenter 301 from the image 300. As illustrated in FIGS. 4A and 4B, theregion 302 corresponds to a region 202R located on the right side asviewed from the fisheye camera 42 in the field of view 202. The ECU 22generates the image 303 in which the distortion is reduced by performingthe distortion reduction processing on the region 302. This image 303 isan image representing the situation of the region 202R.

As a result of the distortion reduction processing, the distortion isreduced at positions closer to the transform center 301, and thedistortion is not reduced or is increased at positions farther from thetransform center 301. In a case where the entire image 300 is a targetof the distortion reduction processing, the distortion increases in aregion located farther from the transform center 301. Thus, even if theexternal environment of the vehicle 1 is analyzed using this regionlocated farther, accurate analysis cannot be performed. Accordingly, thecontrol device 2 sets the transform center 301 in the analysis targetregion, performs distortion reduction processing on the region aroundthe transform center 301, and analyzes the situation of the analysistarget region using the processed image.

The field of view 201 includes, as analysis target regions, a region201L for capturing in the direction left-diagonally ahead of the vehicle1, a region 201F for capturing in the direction directly ahead of thevehicle 1, and a region 201R for capturing in the directionright-diagonally ahead of the vehicle 1. The field of view 202 includes,as analysis target regions, a region 202L for capturing in the directionright-diagonally ahead of the vehicle 1, a region 202F for capturing inthe direction directly right of the vehicle 1, and a region 202R forcapturing in the direction right-diagonally behind of the vehicle 1. Thefield of view 203 includes, as analysis target regions, a region 203Lfor capturing in the direction right-diagonally behind of the vehicle 1,a region 203F for capturing in the direction directly behind of thevehicle 1, and a region 203R for capturing in the directionleft-diagonally behind of the vehicle 1. The field of view 204 includes,as analysis target regions, a region 204L for capturing in the directionleft-diagonally behind of the vehicle 1, a region 204F for capturing inthe direction directly left of the vehicle 1, and a region 204R forcapturing in the direction left-diagonally ahead of the vehicle 1. Thefield of view 201 may be evenly (that is, such that the angles of viewin the horizontal direction of the respective regions are equal) dividedinto the three regions 201L, 201F, and 201R in the horizontal direction.The other fields of view 202 to 204 may also be evenly divided intothree.

In a case where it is desired to analyze the situation in the directionright-diagonally ahead of the vehicle 1, the control device 2 sets thetransform center 301 in the region 202L (for example, the center of theregion 202L) included in the field of view 202 of the fisheye camera 42,performs the distortion reduction processing on a region around thetransform center 301, and analyzes the situation in the directionright-diagonally ahead of using the processed image. In a case where itis desired to analyze the situation in the direction directly right ofthe vehicle 1, the control device 2 sets the transform center 301 in theregion 202F (for example, the center of the region 202F) included in thefield of view 202 of the fisheye camera 42, performs the distortionreduction processing on a region around the transform center 301, andanalyzes the situation in the direction directly right of the vehicle 1using the processed image. In a case where it is desired to analyze thesituation in the direction right-diagonally behind of the vehicle 1, thecontrol device 2 sets the transform center 301 in the region 202R (forexample, the center of the region 202R) included in the field of view202 of the fisheye camera 42, performs the distortion reductionprocessing on a region around the transform center 301, and analyzes thesituation in the direction right-diagonally behind of the vehicle 1using the processed image.

An example of a method in which the control device 2 controls thevehicle 1 in some embodiments will be described with reference to FIG.5. This method may be performed by the processor 20 a of each of theECUs 20 to 29 of the control device 2 executing a program in the memory20 b. The method of FIG. 5 may be started in response to turning on of adriving assistance function or an automated driving function by thecontrol device 2.

In step S501, the control device 2 acquires an image of the externalenvironment of the vehicle 1 from each of the standard camera 40 and thefisheye cameras 41 to 44. Each image includes the situation of theranges described in FIGS. 2A to 2D in the external environment of thevehicle 1.

In step S502, the control device 2 determines the current travelingscene of the vehicle 1. In the example described below, as the travelingscene of the vehicle, (1) a scene where the vehicle enters a T-junctionor a scene where the vehicle restarts after a temporary stop, (2) ascene where the vehicle travels on a narrow road, (3) a scene where thevehicle turns right or left at the intersection, (4) a scene where thevehicle moves backward, and (5) a scene where the vehicle changes lanesare handled. Scenes other than these are handled as a normal (default)scene. The normal scene includes, for example, a scene where the vehicle1 is traveling along a road. In the present specification, cases wherethe vehicle 1 travels in a country where left-hand traffic is employedwill be handled. In a country where right-hand traffic is employed, leftturn and right turn in the following description are interchanged.

In step S503, the control device 2 determines one or more regions to betargets of the distortion reduction processing in the image acquired instep S501 according to the rule corresponding to the current travelingscene of the vehicle 1. Hereinafter, this rule will be referred to as aregion determination rule. The region determination rule is determinedin advance and stored in, for example, the memory 20 b. A specificexample of the region determination rule will be described later.

In step S504, as illustrated in FIG. 3, the control device 2 performsthe distortion reduction processing for each of one or more regionsdetermined as targets of the distortion reduction processing. Thisdistortion reduction processing is processing for reducing distortion ofimages acquired from the fisheye cameras 41 to 44. Since an existingtechnique may be used for the distortion reduction processing, detaileddescription thereof will be omitted. It is not necessary to perform thedistortion reduction processing on an image acquired from the standardcamera 40.

In step S505, the control device 2 recognizes the external environmentof the vehicle 1 based on the image acquired from the standard camera 40and the images acquired from the fisheye cameras 41 to 44 and subjectedto the distortion reduction processing. For example, the control device2 may specify the target object around the vehicle 1 by applying thecorrected image to a model learned in advance and stored in the memory20 b. Further, the control device 2 may control (for example, automaticbraking, notification to the driver, change of automated driving level,and the like) the vehicle 1 according to a recognition result of theexternal environment. Since an existing technique may be applied tocontrol of the vehicle 1 according to the recognition result of theexternal environment, a detailed description thereof will be omitted.

In step S506, the control device 2 determines whether to end theoperation. In a case where it is determined that the operation is to beended (“YES” in step S506), the control device 2 ends the operation, andotherwise (“NO” in step S506), the control device 2 returns theoperation to step S501. The control device 2 may determine to end theoperation, for example, in response to turning off of the drivingassistance function or the automated driving function.

As described above, steps S501 to S505 are repeatedly executed. Thecontrol device 2 may cyclically execute the operations of steps S501 toS505. This execution cycle varies depending on the time required for thedistortion reduction processing in S504 and recognition processing inS505, and may be, for example, about 100 ms.

The cyclic operation of the control device 2 will be described withreference to FIG. 6. A circle in FIG. 6 indicates an image candidate tobe used in the recognition processing in step S505. At in FIG. 6indicates a cycle in which steps S501 to S505 are executed. The imageacquired from the standard camera 40 has little distortion, and thus canbe used in the recognition processing of step S505 without performingthe distortion reduction processing of step S504. The image acquiredfrom the fisheye cameras 41 to 44 is used in the recognition processingof step S505 after the distortion reduction processing is performed instep S504. As described above, the image acquired from the fisheyecamera can generate an image in which the distortion reductionprocessing is performed on each of the three regions divided in thehorizontal direction. Thus, at one operation timing of the cyclicoperation, the control device 2 can execute the recognition processingusing a maximum of 13 images. Twelve of the 13 images are acquired fromthe fisheye camera, and thus the distortion reduction processing isperformed before the recognition processing. When the distortionreduction processing is performed on all of these 12 regions, theprocessing load increases and the power consumption also increases.Accordingly, in the following embodiment, it is determined which of the12 regions is subjected to the distortion reduction processing at eachoperation timing and used for the recognition processing based on thetraveling scene of the vehicle 1.

The region determination rule of the normal scene will be described withreference to FIGS. 7 to 9C. FIG. 7 illustrates an example of a travelingscene. In the example illustrated in FIG. 7, the vehicle 1 is travelingalong a straight road.

FIG. 8 illustrates regions to be analysis targets at respectiveoperation timings. The control device 2 repeats states 800 to 802 inorder. That is, assuming that the state is the state 800 at theoperation timing of time t, the state becomes the state 801 at theoperation timing of time t+Δt (as described above, Δt indicates acycle), the state becomes the state 802 at the operation timing of timet+2×Δt, and the state returns to the state 800 at the operation timingof time t+3×Δt. The same applies to a region determination rule in othertraveling scenes described later.

In the state 800, the field of view 200 by the standard camera 40, theregion 201L by the fisheye camera 41, the region 202R by the fisheyecamera 42, the region 203L by the fisheye camera 43, and the region 204Rby the fisheye camera 44 are to be analysis targets. In the state 801,the field of view 200 by the standard camera 40, the region 201F by thefisheye camera 41, the region 202F by the fisheye camera 42, the region203F by the fisheye camera 43, and the region 204F by the fisheye camera44 are to be analysis targets. In the state 802, the field of view 200by the standard camera 40, the region 201R by the fisheye camera 41, theregion 202L by the fisheye camera 42, the region 203R by the fisheyecamera 43, and the region 204L by the fisheye camera 44 are to beanalysis targets.

In a case where a region included in the field of view of a fisheyecamera is to be an analysis target, the control device 2 performs thedistortion reduction processing on this region as described above.Therefore, the region determination rule defines the position in thehorizontal direction of the region to be a target of the distortionreduction processing and the timing at which the region at this positionis set as a target of the distortion reduction processing. In addition,the region determination rule individually defines a rule for each ofthe plurality of fisheye cameras 41 to 44.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1as an analysis target every cycle (that is, every time), and sets eachof the direction right-diagonally ahead of the vehicle 1, the directiondirectly right of the vehicle 1, the direction right-diagonally behindof the vehicle 1, the direction directly behind of the vehicle 1, thedirection left-diagonally behind of the vehicle 1, the directiondirectly left of the vehicle 1, and the direction left-diagonally aheadof the vehicle 1 as an analysis target every three cycles. In addition,regions not to be analysis targets are distributed to a plurality ofoperation timings so that loads are not concentrated on the controldevice 2 at a specific operation timing. Further, analysis using boththe image of the standard camera 40 and the image of a fisheye camera isperformed every three cycles for the direction directly ahead of thevehicle 1, and analysis using both the images of two fisheye cameras isperformed every three cycles for each of the four diagonal directions ofthe vehicle 1. In this manner, by setting a part of the images of thefisheye cameras 41 to 44 on which the distortion correction processingis performed as an analysis target at each operation timing, theprocessing load of the control device 2 is reduced, and the powerconsumption is reduced.

FIGS. 9A to 9C illustrate positions in the vertical direction ofanalysis target regions. FIG. 9A illustrates the position in thevertical direction of the region 201F by the fisheye camera 41. FIG. 9Billustrates the position in the vertical direction of the region 202F bythe fisheye camera 42. FIG. 9C illustrates the position in the verticaldirection of the region 203F by the fisheye camera 43. The position inthe vertical direction of each region by the fisheye camera 44 may besimilar to that in FIG. 9B, and thus the description thereof will beomitted.

In the region determination rule, as illustrated in FIG. 9A, an angleformed by the transform center 301 defining the region 201F and avertically downward direction of the vehicle 1 is defined as θ1. θ1 maybe 90 degrees or a value smaller than 90 degrees (for example, 80degrees). Also in the regions 201R and 201L, the angle formed by thetransform center 301 and the vertically downward direction of thevehicle 1 may be defined as θ1. Similarly, for the regions of thefisheye cameras 42 to 44, the angle formed by the transform center 301and the vertically downward direction of the vehicle 1 may be defined asθ1. As described above, the region determination rule defines theposition in the vertical direction of the region to be a target of thedistortion reduction processing. By setting the angle formed by thetransform center 301 and the vertically downward direction of thevehicle 1 to θ1 (for example, 90 degrees), a distant place and thevicinity of the vehicle 1 can be analyzed in a well-balanced manner.

With reference to FIGS. 10 to 12, the region determination rule of thescene where the vehicle 1 enters a T-junction or the scene where thevehicle 1 restarts after a temporary stop will be described. FIG. 10illustrates an example of such a scene. In the example illustrated inFIG. 10, the vehicle 1 is about to enter a T-junction. There is a stopsign ahead of the T-junction, and thus the vehicle 1 pauses here andrestarts. The example of FIG. 10 illustrates a scene where both entry toa T-junction and restart after a temporary stop are performed, but thefollowing region determination rule may be applied to a scene where onlyone of these is performed. The positions in the vertical direction ofthe analysis target regions of the scene where the vehicle 1 enters theT-junction or the scene where the vehicle 1 restarts after a temporarystop may be similar to those in the description in FIGS. 9A to 9C, andthus redundant description will be omitted.

FIG. 11 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. Thecontrol device 2 repeats a state 1100. In the state 1100, the field ofview 200 by the standard camera 40, the regions 201L and 201R by thefisheye camera 41, the region 202L by the fisheye camera 42, and theregion 204R by the fisheye camera 44 are to be analysis targets. In thisexample, the control device 2 sets each of the direction directly aheadof the vehicle 1, the direction right-diagonally ahead of the vehicle 1,and the direction left-diagonally ahead of the vehicle 1 as an analysistarget every cycle (that is, every time). On the other hand, thedirection directly right of the vehicle 1, the directionright-diagonally behind of the vehicle 1, the direction directly behindof the vehicle 1, the direction left-diagonally behind of the vehicle 1,and the direction directly left of the vehicle 1 are not to be analysistargets. In addition, analysis using both the images of two fisheyecameras is performed every time for the direction right-diagonally aheadof the vehicle 1 and the direction left-diagonally ahead of the vehicle1. By performing analysis using images from two different fisheyecameras, distance measurement by stereo vision can be executed, and thusanalysis with higher accuracy can be performed.

FIG. 12 illustrates regions to be analysis targets at respectiveoperation timings in another example of the region determination rule.The control device 2 repeats states 1200 to 1202 in order. In the state1200, the field of view 200 by the standard camera 40, the regions 201Land 201R by the fisheye camera 41, and the regions 203L and 203R by thefisheye camera 43 are to be analysis targets. In the state 1201, thefield of view 200 by the standard camera 40, the regions 201L and 201Rby the fisheye camera 41, the region 202F by the fisheye camera 42, andthe region 204F by the fisheye camera 44 are to be analysis targets. Inthe state 1202, the field of view 200 by the standard camera 40, theregions 201L and 201R by the fisheye camera 41, the region 202L by thefisheye camera 42, and the region 204R by the fisheye camera 44 are tobe analysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1,the direction right-diagonally ahead of the vehicle 1, and the directionleft-diagonally ahead of the vehicle 1 as analysis targets every cycle(that is, every time), and sets each of the direction directly right ofthe vehicle 1, the direction right-diagonally behind of the vehicle 1,the direction directly behind of the vehicle 1, the directionleft-diagonally behind of the vehicle 1, and the direction directly leftof the vehicle 1 as an analysis target every three cycles. In addition,regions not to be analysis targets are distributed to a plurality ofoperation timings so that loads are not concentrated on the controldevice 2 at a specific operation timing. Further, analysis using boththe images of two fisheye cameras is performed every three cycles forthe direction right-diagonally ahead of the vehicle 1 and the directionleft-diagonally ahead of the vehicle 1.

In any of the above examples, the region determination rule defines thatthe direction left-diagonally ahead of the vehicle 1 and the directionright-diagonally ahead of the vehicle 1 are set as targets of thedistortion reduction processing more frequently than the directiondirectly right of the vehicle 1, the direction directly left of thevehicle 1, the direction right-diagonally behind of the vehicle 1, thedirection left-diagonally behind of the vehicle 1, and the directiondirectly behind of the vehicle 1. In the scene where the vehicle 1enters the T-junction or the scene where the vehicle 1 restarts after atemporary stop, there is a high possibility that other trafficparticipants (pedestrian, bicycle, or another vehicle) appear from thedirection diagonally ahead of the vehicle 1. Accordingly, by setting thedirection left-diagonally ahead of the vehicle 1 and the directionright-diagonally ahead of the vehicle 1 as analysis targets with highfrequency, it is possible to execute appropriate analysis according tothe traveling scene while reducing the processing load of the controldevice 2.

With reference to FIGS. 13A to 16C, the region determination rule of thescene where the vehicle 1 travels on a narrow road will be described.The narrow road may be a traveling scene where the distance between thevehicle 1 and the obstacle is equal to or less than a threshold (forexample, 50 cm or less). FIGS. 13A to 13D illustrate an example of sucha scene. FIG. 13A illustrates a scene where the vehicle 1 travels on anS-shaped curve. FIG. 13B illustrates a scene where the vehicle 1 travelson an L-shaped road. FIG. 13C illustrates a scene where the vehicle 1passes an oncoming vehicle. FIG. 13D illustrates a scene where thevehicle 1 passes by a preceding vehicle that is about to turn right andtravels. As illustrated in FIGS. 13A and 13B, a narrow road may occuraccording to a road shape, and as illustrated in FIGS. 13C and 13D, anarrow road may occur according to a traffic situation.

FIG. 14 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. Thecontrol device 2 repeats states 1400 to 1406 in order. In the state1400, the field of view 200 by the standard camera 40, the region 201Fby the fisheye camera 41, the regions 202L and 202F by the fisheyecamera 42, and the region 204R by the fisheye camera 44 are to beanalysis targets. In the state 1401, the field of view 200 by thestandard camera 40, the region 201F by the fisheye camera 41, theregions 202L and 202R by the fisheye camera 42, and the region 204R bythe fisheye camera 44 are to be analysis targets. In the state 1402, thefield of view 200 by the standard camera 40, the region 201F by thefisheye camera 41, the region 202L by the fisheye camera 42, the region203L by the fisheye camera 43, and the region 204R by the fisheye camera44 are to be analysis targets. In the state 1403, the field of view 200by the standard camera 40, the region 201F by the fisheye camera 41, theregion 202L by the fisheye camera 42, the region 203F by the fisheyecamera 43, and the region 204R by the fisheye camera 44 are to beanalysis targets. In the state 1404, the field of view 200 by thestandard camera 40, the region 201F by the fisheye camera 41, the region202L by the fisheye camera 42, the region 203R by the fisheye camera 43,and the region 204R by the fisheye camera 44 are to be analysis targets.In the state 1405, the field of view 200 by the standard camera 40, theregion 201F by the fisheye camera 41, the region 202L by the fisheyecamera 42, and the regions 204R and 204L by the fisheye camera 44 are tobe analysis targets. In the state 1406, the field of view 200 by thestandard camera 40, the region 201F by the fisheye camera 41, the region202L by the fisheye camera 42, and the regions 204R and 204F by thefisheye camera 44 are to be analysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1,the direction right-diagonally ahead of the vehicle 1, and the directionleft-diagonally ahead of the vehicle 1 as analysis targets every cycle(that is, every time), sets each of the direction directly right of thevehicle 1, the direction directly behind of the vehicle 1, and thedirection directly left of the vehicle 1 as an analysis target everyseven cycles, and sets each of the direction right-diagonally behind ofthe vehicle 1 and the direction left-diagonally behind of the vehicle 1as an analysis target twice among seven cycles. In addition, regions notto be analysis targets are distributed to a plurality of operationtimings so that loads are not concentrated on the control device 2 at aspecific operation timing.

FIG. 15 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. Thecontrol device 2 repeats states 1500 to 1502 in order. In the state1500, the field of view 200 by the standard camera 40, the region 201Fby the fisheye camera 41, the region 202L by the fisheye camera 42, theregion 203F by the fisheye camera 43, and the region 204R by the fisheyecamera 44 are set as analysis targets. In the state 1501, the field ofview 200 by the standard camera 40, the regions 202L and 202F by thefisheye camera 42, and the regions 204F and 204R by the fisheye camera44 are set as analysis targets. In the state 1502, the field of view 200by the standard camera 40, the region 202L by the fisheye camera 42, theregions 203L and 203R by the fisheye camera 43, and the region 204R bythe fisheye camera 44 are set as analysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1,the direction right-diagonally ahead of the vehicle 1, and the directionleft-diagonally ahead of the vehicle 1 as analysis targets every cycle(that is, every time), and sets each of the direction directly right ofthe vehicle 1, the direction right-diagonally behind of the vehicle 1,the direction directly behind of the vehicle 1, the directionleft-diagonally behind of the vehicle 1, and the direction directly leftof the vehicle 1 as an analysis target every three cycles. In addition,regions not to be analysis targets are distributed to a plurality ofoperation timings so that loads are not concentrated on the controldevice 2 at a specific operation timing.

FIGS. 16A to 16C illustrate positions in the vertical direction of theanalysis target regions. FIG. 16A illustrates the position in thevertical direction of the region 201F by the fisheye camera 41. FIG. 16Billustrates the position in the vertical direction of the region 202F bythe fisheye camera 42. FIG. 16C illustrates the position in the verticaldirection of the region 203F by the fisheye camera 43. The position inthe vertical direction of each region by the fisheye camera 44 may besimilar to that in FIG. 16B, and thus the description thereof will beomitted.

In the region determination rule, as illustrated in FIG. 16A, an angleformed by the transform center 301 defining the region 201F and thevertically downward direction of the vehicle 1 is defined as θ2. θ2 is avalue smaller than θ1 in FIGS. 9A to 9C, and may be, for example, 70degrees. Also in the regions 201R and 201L, the angle formed by thetransform center 301 and the vertically downward direction of thevehicle 1 may be defined as θ2. Also in the regions of the fisheyecameras 42 to 44, the angle formed by the transform center 301 and thevertically downward direction of the vehicle 1 is defined as θ3. θ3 is avalue smaller than θ2, and may be, for example, 45 degrees.

As described above, in any of the regions in the direction directlyahead of the vehicle 1, in the direction right-diagonally ahead of thevehicle 1, in the direction directly right of the vehicle 1, in thedirection right-diagonally behind of the vehicle 1, in the directiondirectly behind of the vehicle 1, in the direction left-diagonallybehind of the vehicle 1, in the direction directly left of the vehicle1, and in the direction left-diagonally ahead of the vehicle 1, theposition in the vertical direction of the analysis target region whenthe vehicle 1 travels on a narrow road is on a lower side (for example,the transform center 301 is downward) than that when the vehicle 1travels on a path other than the narrow road (for example, in the caseof the normal scene described above). When the vehicle 1 travels on anarrow road, there is a possibility that a wheel of the vehicle 1 runson a curbstone or falls into a side groove. The analysis accuracy of thesituation near the ground is improved by positioning the analysis targetregion on the lower side. In addition, in the region determination rulewhen the vehicle 1 travels on a narrow road, the region 201F in thedirection directly ahead of the vehicle 1 by the fisheye camera 41 is ananalysis target. Thus, as illustrated in FIG. 16A, it is possible toanalyze the vicinity in the direction directly ahead of the vehicle 1that is not included in the field of view 200 of the standard camera 40.In addition, the situation in the direction right-diagonally ahead ofthe vehicle 1 is analyzed based on the region 202L by the fisheye camera42. Thus, it is possible to analyze a region near the front wheel of thevehicle 1. The same applies to the situation in the directionleft-diagonally ahead of the vehicle 1.

When the vehicle 1 travels on a narrow road, it is less necessary toanalyze the direction left-diagonally and right-diagonally ahead of thevehicle for a far distance because an obstacle is near, but it is betterto analyze the direction ahead of the vehicle 1 (including the directionright-diagonally ahead of the vehicle 1 and in the directionleft-diagonally ahead of the vehicle 1) up to a certain distance.Therefore, in the above example, the position in the vertical directionof the region including the direction right-diagonally ahead of thevehicle 1 and the direction left-diagonally ahead of the vehicle 1 isset to a higher side than the position in the vertical direction of theregion including the direction directly right of the vehicle 1 and thedirection directly left of the vehicle 1 (that is, θ2>θ3).

In any of the above examples, the region determination rule defines thatthe direction right-diagonally ahead of the vehicle 1 and the directionleft-diagonally ahead of the vehicle 1 are set as targets of thedistortion reduction processing more frequently than the directiondirectly right of the vehicle 1, the direction directly left of thevehicle 1, the direction right-diagonally behind of the vehicle 1, thedirection left-diagonally behind of the vehicle 1, and the directiondirectly behind of the vehicle 1. In the scene where the vehicle 1travels on a narrow road, there is a high possibility that the vehicle 1comes into contact with objects in the direction left-diagonally aheadof the vehicle 1 or in the direction right-diagonally ahead of thevehicle 1. Accordingly, by setting the direction left-diagonally aheadof the vehicle 1 and the direction right-diagonally ahead of the vehicle1 as analysis targets with high frequency, it is possible to executeappropriate analysis according to the traveling scene while reducing theprocessing load of the control device 2.

The region determination rule of the scene where the vehicle 1 turns atan intersection will be described with reference to FIGS. 17 to 20.FIGS. 17 and 19 illustrate examples of such scenes. In the exampleillustrated in FIG. 17, the vehicle 1 turns left at the intersection. Ina country of left-hand traffic, left turn means that the vehicle 1 turnsat an intersection in a direction not intersecting with an oppositelane. In the example illustrated in FIG. 19, the vehicle 1 turns rightat the intersection. In a country of left-hand traffic, right turn meansthat the vehicle 1 turns at an intersection in a direction intersectingwith an opposite lane. The position in the vertical direction of theanalysis target region of the scene where the vehicle 1 turns at theintersection may be similar to those in the description in FIGS. 9A to9C, and thus redundant description will be omitted.

FIG. 18 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule of ascene where the vehicle 1 turns left at an intersection. The controldevice 2 repeats states 1800 to 1802 in order. In the state 1800, thefield of view 200 by the standard camera 40, the region 201L by thefisheye camera 41, the region 202R by the fisheye camera 42, the region203R by the fisheye camera 43, and the region 204F by the fisheye camera44 are to be analysis targets. In the state 1801, the field of view 200by the standard camera 40, the region 201L by the fisheye camera 41, theregion 202F by the fisheye camera 42, the regions 203F and 203R by thefisheye camera 43, and the region 204F by the fisheye camera 44 are tobe analysis targets. In the state 1802, the field of view 200 by thestandard camera 40, the region 201L by the fisheye camera 41, the region202L by the fisheye camera 42, the regions 203L and 203R by the fisheyecamera 43, and the region 204F by the fisheye camera 44 are to beanalysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1,the direction left-diagonally ahead of the vehicle 1, the directiondirectly left of the vehicle 1, and the direction left-diagonally behindof the vehicle 1 as analysis targets every cycle (that is, every time),and sets each of the direction right-diagonally ahead of the vehicle 1,the direction directly right of the vehicle 1, the directionright-diagonally behind of the vehicle 1, and the direction directlybehind of the vehicle 1 as an analysis target every three cycles. Inaddition, regions not to be analysis targets are distributed to aplurality of operation timings so that loads are not concentrated on thecontrol device 2 at a specific operation timing.

FIG. 20 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule of ascene where the vehicle 1 turns right at an intersection. The controldevice 2 repeats the states 2000 to 2002 in order. In the state 2000,the field of view 200 by the standard camera 40, the regions 201F and201R by the fisheye camera 41, the region 203L by the fisheye camera 43,and the region 204F by the fisheye camera 44 are to be analysis targets.In the state 2001, the field of view 200 by the standard camera 40, theregions 201F and 201R by the fisheye camera 41, the region 203F by thefisheye camera 43, and the region 204R by the fisheye camera 44 are tobe analysis targets. In the state 2002, the field of view 200 by thestandard camera 40, the regions 201F and 201R by the fisheye camera 41,the region 202F by the fisheye camera 42, and the region 203R by thefisheye camera 43 are to be analysis targets.

As described above, by transitioning the state at every operationtiming, the control device 2 sets the direction directly ahead of thevehicle 1 and the direction right-diagonally ahead of the vehicle 1 asthe analysis target every cycle (that is, every time), and sets each ofthe direction directly right of the vehicle 1, the directionright-diagonally behind of the vehicle 1, the direction directly behindof the vehicle 1, the direction left-diagonally behind of the vehicle 1,the direction directly left of the vehicle 1, and the directionleft-diagonally ahead of the vehicle 1 as an analysis target every threecycles. In addition, regions not to be analysis targets are distributedto a plurality of operation timings so that loads are not concentratedon the control device 2 at a specific operation timing.

In any of the above examples, the region determination rule defines thatthe direction diagonally ahead of the vehicle 1 and on the same side ofthe direction to which the vehicle 1 turns is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle 1 and the direction left or right of thevehicle 1 (including the direction diagonally ahead of the vehicle 1,the direction directly left or right of the vehicle 1, and the directiondiagonally behind of the vehicle 1) and on the opposite side of thedirection to which the vehicle 1 turns. In a scene where the vehicle 1turns at an intersection, there is a high possibility that anothertraffic participant (pedestrian, bicycle, or another vehicle) appearsfrom the direction diagonally ahead of the vehicle 1 and on the sameside of the direction to which the vehicle 1 turns. Accordingly, bysetting the direction diagonally ahead of the vehicle 1 and on the sameside of the direction to which the vehicle 1 turns as the analysistarget with high frequency, it is possible to execute appropriateanalysis according to the traveling scene while reducing the processingload of the control device 2.

In the above example, when the vehicle 1 turns left, the directionleft-diagonally behind of the vehicle 1 is set as a target of thedistortion reduction processing more frequently than the direction leftor right of the vehicle 1 (including the direction diagonally ahead ofthe vehicle 1, the direction directly left or right of the vehicle 1,and the direction diagonally behind of the vehicle 1) and on an oppositeside of the direction to which the vehicle 1 turns. By setting thedirection left-diagonally behind of the vehicle 1 as an analysis targetwith high frequency in this manner, the recognition accuracy of thetraffic participant that causes a winding accident when turning left isimproved. The region determination rule defines that, when the vehicle 1turns right, the direction directly ahead of the vehicle 1 is set as atarget of the distortion reduction processing more frequently than thedirection left or right of the vehicle 1 (including the directiondiagonally ahead of the vehicle 1, the direction directly left or rightof the vehicle 1, and the direction diagonally behind of the vehicle 1)and on an opposite side of the direction to which the vehicle 1 turns.By setting the direction directly ahead of the vehicle 1 as an analysistarget with high frequency in this manner, the recognition accuracy ofthe oncoming vehicle when turning right is improved.

The region determination rule of the scene where the vehicle 1 movesbackward will be described with reference to FIGS. 21 to 23. FIG. 21illustrates an example of such a scene. In the example illustrated inFIG. 21, the vehicle 1 is about to move backward to start from a parkingspace. The position in the vertical direction of the analysis targetregion of the scene where the vehicle 1 moves backward may be similar tothose in the description in FIGS. 16A to 16C, and thus redundantdescription will be omitted.

FIG. 22 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. Thecontrol device 2 repeats states 2200 to 2204 in order. In the state2200, the field of view 200 by the standard camera 40, the regions 203L,203F, and 203R by the fisheye camera 43, and the region 204F by thefisheye camera 44 are to be analysis targets. In the state 2201, thefield of view 200 by the standard camera 40, the region 201L by thefisheye camera 41, and the regions 203L, 203F, and 203R by the fisheyecamera 43 are to be analysis targets. In the state 2202, the field ofview 200 by the standard camera 40, the region 201F by the fisheyecamera 41, and the regions 203L, 203F, and 203R by the fisheye camera 43are to be analysis targets. In the state 2203, the field of view 200 bythe standard camera 40, the region 201R by the fisheye camera 41, andthe regions 203L, 203F, and 203R by the fisheye camera 43 are to beanalysis targets. In the state 2204, the field of view 200 by thestandard camera 40, the region 202F by the fisheye camera 42, and theregions 203L, 203F, and 203R by the fisheye camera 43 are to be analysistargets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1,the direction right-diagonally behind of the vehicle 1, the directiondirectly behind of the vehicle 1, and the direction left-diagonallybehind of the vehicle 1 as analysis targets every cycle (that is, everytime), and sets each of the direction left-diagonally ahead of thevehicle 1, the direction right-diagonally ahead of the vehicle 1, thedirection directly right of the vehicle 1, the direction directly leftof the vehicle 1, and the direction left-diagonally ahead of the vehicle1 as analysis targets every five cycles. In addition, regions not to beanalysis targets are distributed to a plurality of operation timings sothat loads are not concentrated on the control device 2 at a specificoperation timing.

FIG. 23 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. Thecontrol device 2 repeats states 2300 to 2205 in order. In the state2300, the field of view 200 by the standard camera 40, the region 201Lby the fisheye camera 41, and the regions 203L, 203F, and 203R by thefisheye camera 43 are to be analysis targets. In the state 2301, thefield of view 200 by the standard camera 40, the region 201F by thefisheye camera 41, the region 202F by the fisheye camera 42, the region203F by the fisheye camera 43, and the region 204F by the fisheye camera44 are to be analysis targets. In the state 2302, the field of view 200by the standard camera 40, the region 201R by the fisheye camera 41, andthe regions 203L, 203F, and 203R by the fisheye camera 43 are to beanalysis targets. In the state 2303, the field of view 200 by thestandard camera 40, the region 201L by the fisheye camera 41, the region202F by the fisheye camera 42, the region 203F by the fisheye camera 43,and the region 204F by the fisheye camera 44 are to be analysis targets.In the state 2304, the field of view 200 by the standard camera 40, theregion 201F by the fisheye camera 41, and the regions 203L, 203F, and203R by the fisheye camera 43 are to be analysis targets. In the state2305, the field of view 200 by the standard camera 40, the region 201Rby the fisheye camera 41, the region 202F by the fisheye camera 42, theregion 203F by the fisheye camera 43, and the region 204F by the fisheyecamera 44 are to be analysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction directly ahead of the vehicle 1and the direction directly behind of the vehicle 1 as analysis targetsevery cycle (that is, every time), sets each of the directionright-diagonally behind of the vehicle 1 and the directionleft-diagonally behind of the vehicle 1 as analysis targets every twocycles, and sets each of the direction right-diagonally ahead of thevehicle 1, the direction directly right of the vehicle 1, the directiondirectly left of the vehicle 1, and the direction left-diagonally aheadof the vehicle 1 as analysis targets every three cycles. In addition,regions not to be analysis targets are distributed to a plurality ofoperation timings so that loads are not concentrated on the controldevice 2 at a specific operation timing.

In any of the above examples, the region determination rule defines thatthe direction directly behind of the vehicle 1, the directionright-diagonally behind of the vehicle 1, and the directionleft-diagonally behind of the vehicle 1 are set as targets of thedistortion reduction processing more frequently than the directionright-diagonally ahead of the vehicle 1 and the directionleft-diagonally ahead of the vehicle 1. In a scene where the vehicle 1moves backward, there is a high possibility that other trafficparticipants (pedestrian, bicycle, or another vehicle) appear from thedirection left-diagonally behind of the vehicle 1, the directiondirectly behind of the vehicle 1, or the direction right-diagonallybehind of the vehicle 1. Accordingly, by setting the direction directlybehind of the vehicle 1, the direction right-diagonally behind of thevehicle 1, and the direction left-diagonally behind of the vehicle 1 asthe analysis targets with high frequency, it is possible to executeappropriate analysis according to the traveling scene while reducing theprocessing load of the control device 2. Further, in the example of FIG.23, the direction directly behind of the vehicle 1 is set as a target ofthe distortion reduction processing more frequently than the directionright-diagonally behind of the vehicle 1 and the directionleft-diagonally behind of the vehicle 1. An image from the standardcamera 40 is set as an analysis target every time in the directiondirectly ahead of the vehicle 1, but such a standard camera does notexist in the rear of the vehicle 1. Accordingly, an image in thedirection directly behind of the vehicle 1 captured by the fisheyecamera 43 may be set as the analysis target with high frequency.

In the above example, the region determination rule defines that, whenthe vehicle 1 moves backward, as compared with cases where the vehicle 1moves forward, the position in the vertical direction of the region inthe direction directly behind of the vehicle 1 is on a lower side. Thus,analysis accuracy of the vicinity in the direction directly behind ofthe vehicle 1, which is a blind spot for the driver, can be improved.

The region determination rule of the scene where the vehicle 1 performsa lane change will be described with reference to FIGS. 24 and 25. FIG.24 illustrates an example of such a scene. The lane change may beperformed according to the travel plan to the destination, or may beperformed to avoid an obstacle (for example, cutting in of anothervehicle or jumping out of a pedestrian) ahead of the vehicle 1.

FIG. 25 illustrates regions to be analysis targets at respectiveoperation timings in an example of the region determination rule. FIG.25 deals with a case where the vehicle 1 moves to the right lane. Whenthe vehicle 1 moves to the left lane, the left and right are reversed.The control device 2 repeats states 2500 to 2505 in order. In the state2500, the field of view 200 by the standard camera 40, the regions 201Land 201R by the fisheye camera 41, the region 202F by the fisheye camera42, and the region 204F by the fisheye camera 44 are to be analysistargets. In the state 2501, the field of view 200 by the standard camera40, the regions 201L and 201R by the fisheye camera 41, and the regions203L and 203R by the fisheye camera 43 are to be analysis targets. Inthe state 2502, the field of view 200 by the standard camera 40, theregions 201L and 201R by the fisheye camera 41, the region 202F by thefisheye camera 42, and the region 203F by the fisheye camera 43 are tobe analysis targets. In the state 2503, the field of view 200 by thestandard camera 40, the regions 201L and 201R by the fisheye camera 41,the region 203L by the fisheye camera 43, and the region 204F by thefisheye camera 44 are to be analysis targets. In the state 2504, thefield of view 200 by the standard camera 40, the regions 201L and 201Rby the fisheye camera 41, the region 202F by the fisheye camera 42, andthe region 203R by the fisheye camera 43 are to be analysis targets. Inthe state 2505, the field of view 200 by the standard camera 40, theregions 201L and 201R by the fisheye camera 41, and the regions 203L and203F by the fisheye camera 43 are to be analysis targets.

By transitioning the state at every operation timing as described above,the control device 2 sets the direction left-diagonally ahead of thevehicle 1, the direction directly ahead of the vehicle 1, and thedirection right-diagonally ahead of the vehicle 1 as the analysistargets every cycle (that is, every time), sets each of the directiondirectly right of the vehicle 1 and the direction right-diagonallybehind of the vehicle 1 as the analysis target every two cycles, andsets each of the direction directly behind of the vehicle 1, thedirection left-diagonally behind of the vehicle 1, and the directiondirectly left of the vehicle 1 as the analysis target every threecycles. In addition, regions not to be analysis targets are distributedto a plurality of operation timings so that loads are not concentratedon the control device 2 at a specific operation timing.

In the above example, the region determination rule defines that thedirection right-diagonally ahead of the vehicle 1 and the directionleft-diagonally ahead of the vehicle 1 are set as targets of thedistortion reduction processing more frequently than the directiondiagonally ahead of the vehicle 1 and on the same side of the movingdirection of the vehicle 1 and the direction diagonally behind of thevehicle 1 and on the same side of the moving direction of the vehicle 1.Further, the region determination rule defines that the directiondiagonally ahead of the vehicle 1 and on the same side of the movingdirection of the vehicle 1 and the direction diagonally behind of thevehicle 1 and on the same side of the moving direction of the vehicle 1are set as targets of the distortion reduction processing morefrequently than the direction diagonally ahead of the vehicle 1 and onthe opposite side of the moving direction of the vehicle 1 and thedirection diagonally behind of the vehicle 1 and on the opposite side ofthe moving direction of the vehicle 1. When the vehicle 1 performs alane change, the direction ahead of the vehicle 1 (including thedirection left-diagonally ahead of the vehicle 1, the direction directlyahead of the vehicle 1, and the direction right-diagonally ahead of thevehicle 1) and on the same side of the traveling direction of thevehicle is set as an analysis target more frequently, and the movingdirection (for example, in a case of movement to the right lane, thedirection directly right of the vehicle 1 and the directionright-diagonally behind of the vehicle 1) of the vehicle 1 is set as ananalysis target more frequently than the opposite side (for example, ina case of movement to the right lane, the direction directly left of thevehicle 1 and the direction left-diagonally behind of the vehicle 1).Thus, it is possible to execute appropriate analysis according to thetraveling scene while reducing the processing load of the control device2.

The position in the vertical direction of the analysis target region ofthe scene where the vehicle 1 performs the lane change may be similar tothose in the description in FIGS. 9A to 9C. Alternatively, the regiondetermination rule may define that the position in the verticaldirection of the region including the direction diagonally ahead of thevehicle 1 and on the opposite side of the moving direction of thevehicle 1 is on a lower side than the position in the vertical directionof a region including the direction diagonally ahead of the vehicle 1and on the same side of the moving direction of the vehicle 1. Forexample, when the vehicle 1 moves to the right lane, the position in thevertical direction of the region 204R by the fisheye camera 44 may be ona lower side than the position in the vertical direction of the region204L by the fisheye camera 42. For example, when a lane change isperformed for emergency avoidance, there is a high possibility that anobject that causes emergency avoidance exists near vehicle 1.Accordingly, by setting the position in the vertical direction of theregion including the direction diagonally ahead of the vehicle 1 and onthe opposite side of the moving direction of the vehicle 1 to the lowerside, this object can be recognized with high accuracy.

Modification examples of the above-described embodiment will bedescribed. The control device 2 determines one or more regions to betargets of the distortion reduction processing based on theline-of-sight direction of the driver of the vehicle 1 in addition tothe rule corresponding to the current traveling scene of the vehicle 1.The control device 2 may determine the line-of-sight direction of thedriver based on, for example, an image captured by a driver monitorcamera (not illustrated) attached to the vehicle 1. Regarding theline-of-sight direction of the driver, it is considered that the driverperforms an action according to the situation even if the control device2 does not perform the recognition processing. Thus, the control device2 may reduce the frequency to be a target of the distortion reductionprocessing with respect to a region in the line-of-sight direction ofthe driver in the external environment of the vehicle 1. For example,when the driver is looking at the direction left-diagonally behind ofthe vehicle 1, the control device 2 may reduce the frequency of usingthe region 201L of the fisheye camera 41 or the region 204R of thefisheye camera 44 as the analysis target.

Summary of Embodiment

<Item 1>

A control device (2) for a mobile object including one or more imagingdevices (41-44), the control device comprising:

an image acquisition unit configured to acquire an image (300) of anexternal environment of the mobile object from the one or more imagingdevices;

a correction unit configured to perform distortion reduction processingfor reducing distortion of an image for each of one or more regions(302) included in an image acquired from the one or more imagingdevices; and

a recognition unit configured to recognize the external environment ofthe mobile object based on an image (303) on which the distortionreduction processing has been performed,

wherein the correction unit is configured to determine the one or moreregions to be a target of the distortion reduction processing inaccordance with a predetermined rule according to a moving scene of themobile object.

According to this item, the external environment of the mobile objectcan be appropriately recognized according to the moving scene.

<Item 2>

The control device according to Item 1, wherein

the correction unit is configured to perform the distortion reductionprocessing every predetermined cycle, and

the predetermined rule is a rule that defines a region to be a target ofthe distortion reduction processing every predetermined cycle.

According to this item, a region of interest can be analyzed at aspecified frequency.

<Item 3>

The control device according to Item 1 or 2, wherein

the predetermined rule defines

-   -   a position in a horizontal direction of a region to be a target        of the distortion reduction processing, and    -   a timing at which a region at the position is set as a target of        the distortion reduction processing.

According to this item, a position of interest in the horizontaldirection can be analyzed at a specified frequency.

<Item 4>

The control device according to any of Items 1-3, wherein thepredetermined rule defines a position in a vertical direction of aregion to be a target of the distortion reduction processing.

According to this item, it is possible to intensively analyze a positionof interest in the vertical direction.

<Item 5>

The control device according to any of Items 1-4, wherein

the mobile object includes a plurality of imaging devices,

the image acquisition unit is configured to acquire an image of theexternal environment of the mobile object from each of the plurality ofimaging devices, and

the predetermined rule defines an individual rule for each of theplurality of imaging devices.

According to this item, it is possible to select and intensively analyzeindividual regions included in fields of view of a plurality of imagingdevices.

<Item 6>

The control device according to Item 5, wherein

the plurality of imaging devices includes

-   -   a first imaging device (41) that captures in a direction        directly ahead of the mobile object, in a direction        right-diagonally ahead of the mobile object, and in a direction        left-diagonally ahead of the mobile object,    -   a second imaging device (43) that captures in a direction        directly behind of the mobile object, in a direction        right-diagonally behind of the mobile object, and in a direction        left-diagonally behind of the mobile object,    -   a third imaging device (42) that captures in a direction        directly right of the mobile object, in the direction        right-diagonally ahead of the mobile object, and in the        direction right-diagonally behind of the mobile object, and    -   a fourth imaging device (44) that captures in a direction        directly left of the mobile object, in the direction        left-diagonally ahead of the mobile object, and in the direction        left-diagonally behind of the mobile object.

According to this item, the periphery of the mobile object can beanalyzed in all directions.

<Item 7>

The control device according to any of Items 1-6, wherein the mobileobject is a vehicle (1), and the moving scene is a traveling scene ofthe vehicle.

According to this item, it is possible to intensively analyze adirection of interest when the vehicle travels.

<Item 8>

The control device according to Item 7, wherein

-   -   the predetermined rule defines that, when the vehicle enters a        T-junction or when the vehicle restarts after a temporary stop,    -   the direction left-diagonally ahead of the vehicle and the        direction right-diagonally ahead of the vehicle are set as        targets of the distortion reduction processing more frequently        than the direction directly right of the vehicle, the direction        directly left of the vehicle, the direction right-diagonally        behind of the vehicle, the direction left-diagonally behind of        the vehicle, and the direction directly behind of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when the vehicle enters the T-junction or when thevehicle restarts after a temporary stop.

<Item 9>

The control device according to Item 7 or 8, wherein

the predetermined rule defines that, when the vehicle enters aT-junction or when the vehicle restarts after a temporary stop,

two regions in a direction diagonally ahead of the vehicle out of twoimages acquired from two imaging devices are set as targets of thedistortion reduction processing at a same timing.

According to this item, it is possible to intensively analyze adirection of interest when the vehicle enters the T-junction or when thevehicle restarts after a temporary stop.

<Item 10>

The control device according to any of Items 7-9, wherein

the predetermined rule defines that, when the vehicle travels on anarrow road,

the direction left-diagonally ahead of the vehicle and the directionright-diagonally ahead of the vehicle are set as targets of thedistortion reduction processing more frequently than the directiondirectly right of the vehicle, the direction directly left of thevehicle, the direction right-diagonally behind of the vehicle, thedirection left-diagonally behind of the vehicle, and the directiondirectly behind of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when traveling on a narrow road.

<Item 11>

The control device according to any of Items 7-10, wherein

the predetermined rule defines that, when the vehicle travels on anarrow road,

the direction right-diagonally ahead of the vehicle and the directionleft-diagonally ahead of the vehicle are set as targets of thedistortion reduction processing more frequently than the directiondirectly right of the vehicle, the direction directly left of thevehicle, the direction right-diagonally behind of the vehicle, thedirection left-diagonally behind of the vehicle, and the directiondirectly behind of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when traveling on a narrow road.

<Item 12>

The control device according to any of Items 7-10, wherein

the predetermined rule defines that, when the vehicle travels on anarrow road, as compared with a case where the vehicle travels on a pathother than the narrow road,

positions in a vertical direction of regions in the direction directlyright of the vehicle, in the direction directly left of the vehicle, inthe direction right-diagonally behind of the vehicle, in the directionleft-diagonally behind of the vehicle, and in the direction directlybehind of the vehicle are on a lower side.

According to this item, it is possible to intensively analyze adirection of interest when traveling on a narrow road.

<Item 13>

The control device according to any of Items 7-11, wherein

the predetermined rule defines that, when the vehicle travels on anarrow road, as compared with a case where the vehicle travels on a pathother than the narrow road,

positions in a vertical direction of regions in the direction directlyahead of the vehicle, in the direction right-diagonally ahead of thevehicle, and in the direction left-diagonally ahead of the vehicle areon a lower side.

According to this item, it is possible to intensively analyze adirection of interest when traveling on a narrow road.

<Item 14>

The control device according to any of Items 7-13, wherein

the predetermined rule defines that, when the vehicle travels on anarrow road,

positions in a vertical direction of regions in the directionright-diagonally ahead of the vehicle and in the directionleft-diagonally ahead of the vehicle are on a higher side than positionsin the vertical direction of regions in the direction directly right ofthe vehicle and in the direction directly left of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when traveling on a narrow road.

<Item 15>

The control device according to any of Items 7-14, wherein

the predetermined rule defines that, when the vehicle turns at anintersection,

a direction diagonally ahead of the vehicle and on a same side of adirection to which the vehicle turns is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle and a direction diagonally behind of thevehicle and on an opposite side of the direction to which the vehicleturns.

According to this item, it is possible to intensively analyze adirection of interest when turning at an intersection.

<Item 16>

The control device according to Item 15, wherein

the predetermined rule defines that, when the vehicle turns to adirection intersecting with an opposite lane at an intersection,

the direction directly ahead of the vehicle is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle and the direction diagonally behind ofthe vehicle and on the opposite side of the direction to which thevehicle turns.

According to this item, it is possible to intensively analyze adirection of interest when turning at an intersection.

<Item 17>

The control device according to Item 15 or 16, wherein

the predetermined rule defines that, when the vehicle turns at anintersection in a direction not intersecting an opposite lane,

a direction diagonally behind of the vehicle and on the same side of thedirection to which the vehicle turns is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle and a direction diagonally behind of thevehicle and on the opposite side of the direction to which the vehicleturns.

According to this item, it is possible to intensively analyze adirection of interest when turning at an intersection.

<Item 18>

The control device according to any of Items 7-17, wherein

the predetermined rule defines that, when the vehicle moves backward,

the direction directly behind of the vehicle, the directionright-diagonally behind of the vehicle, and the directionleft-diagonally behind of the vehicle are set as targets of thedistortion reduction processing more frequently than the directionright-diagonally ahead of the vehicle and the direction left-diagonallyahead of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when moving backward.

<Item 19>

The control device according to Item 18, wherein

the predetermined rule is that, when the vehicle moves backward,

the direction directly behind of the vehicle is set as a target of thedistortion reduction processing more frequently than the directionright-diagonally behind of the vehicle and the direction left-diagonallybehind of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest when moving backward.

<Item 20>

The control device according to any of Items 7-19, wherein

the predetermined rule defines that, when the vehicle moves backward, ascompared with a case where the vehicle moves forward,

positions in a vertical direction of regions in the direction directlybehind of the vehicle are on a lower side.

According to this item, it is possible to intensively analyze adirection of interest when moving backward.

<Item 21>

The control device according to any of Items 7-20, wherein

the predetermined rule defines that, when the vehicle performs a lanechange,

the direction right-diagonally ahead of the vehicle and the directionleft-diagonally ahead of the vehicle are set as targets of thedistortion reduction processing more frequently than a directiondirectly left or right of the vehicle and on a same side of a movingdirection of the vehicle and a direction diagonally behind of thevehicle and on the same side of the moving direction the vehicle, and

the direction directly left or right of the vehicle and on the same sideof the moving direction of the vehicle and the direction diagonallybehind of the vehicle and on the same side of the moving direction thevehicle are set as targets of the distortion reduction processing morefrequently than a direction directly left or right of the vehicle and onan opposite side of the moving direction of the vehicle and a directiondiagonally behind of the vehicle and on the opposite side of the movingdirection the vehicle.

According to this item, it is possible to intensively analyze adirection of interest at the time of lane change.

<Item 22>

The control device according to any of Items 7-21, wherein

the predetermined rule defines that, when the vehicle performs a lanechange,

positions in a vertical direction of regions in a direction diagonallyahead of the vehicle and on an opposite side of a moving direction ofthe vehicle is on a lower side than positions in the vertical directionof regions in a direction diagonally ahead of the vehicle and on a sameside of the moving direction of the vehicle.

According to this item, it is possible to intensively analyze adirection of interest at the time of lane change.

<Item 23>

The control device according to any of Items 1-22, wherein each of theone or more imaging devices is an imaging device to which a fisheye lensis attached.

According to this item, the field of view of the imaging device can bewidened.

<Item 24>

The control device according to any of Items 1-23, wherein

the mobile object further includes another imaging device (40) thatcaptures an image with less distortion than the one or more imagingdevices,

the image acquisition unit is configured to acquire an image of theexternal environment of the mobile object from the another imagingdevice, and

the recognition unit is configured to recognize the external environmentof the mobile object further based on an image from the another imagingdevice.

According to this item, the external environment can be recognized usinga plurality of types of imaging devices.

<Item 25>

A vehicle comprising the control device according to any of Items 1-24.

According to this item, the above effect can be obtained in the form ofa vehicle.

<Item 26>

A program for causing a computer to function as each unit of the controldevice according to any of Items 1-24.

According to this item, the above effect can be obtained in the form ofa program.

<Item 27>

A method for controlling a mobile object (1) including one or moreimaging devices (41-44), the method comprising:

acquiring an image (300) of an external environment of the mobile objectfrom the one or more imaging devices;

performing distortion reduction processing for reducing distortion of animage for each of one or more regions (302) included in an imageacquired from the one or more imaging devices; and

recognizing the external environment of the mobile object based on animage (303) on which the distortion reduction processing has beenperformed,

the one or more regions to be a target of the distortion reductionprocessing is determined in accordance with a predetermined ruleaccording to a moving scene of the mobile object.

According to this item, the external environment of the mobile objectcan be appropriately recognized according to the moving scene.

The invention is not limited to the foregoing embodiments, and variousvariations/changes are possible within the spirit of the invention.

What is claimed is:
 1. A control device for a mobile object includingone or more imaging devices, the control device comprising: an imageacquisition unit configured to acquire an image of an externalenvironment of the mobile object from the one or more imaging devices; acorrection unit configured to perform distortion reduction processingfor reducing distortion of an image for each of one or more regionsincluded in an image acquired from the one or more imaging devices; anda recognition unit configured to recognize the external environment ofthe mobile object based on an image on which the distortion reductionprocessing has been performed, wherein the correction unit is configuredto determine the one or more regions to be a target of the distortionreduction processing in accordance with a predetermined rule accordingto a moving scene of the mobile object.
 2. The control device accordingto claim 1, wherein the correction unit is configured to perform thedistortion reduction processing every predetermined cycle, and thepredetermined rule is a rule that defines a region to be a target of thedistortion reduction processing every predetermined cycle.
 3. Thecontrol device according to claim 1, wherein the predetermined ruledefines a position in a horizontal direction of a region to be a targetof the distortion reduction processing, and a timing at which a regionat the position is set as a target of the distortion reductionprocessing.
 4. The control device according to claim 1, wherein thepredetermined rule defines a position in a vertical direction of aregion to be a target of the distortion reduction processing.
 5. Thecontrol device according to claim 1, wherein the mobile object includesa plurality of imaging devices, the image acquisition unit is configuredto acquire an image of the external environment of the mobile objectfrom each of the plurality of imaging devices, and the predeterminedrule defines an individual rule for each of the plurality of imagingdevices.
 6. The control device according to claim 5, wherein theplurality of imaging devices includes a first imaging device thatcaptures in a direction directly ahead of the mobile object, in adirection right-diagonally ahead of the mobile object, and in adirection left-diagonally ahead of the mobile object, a second imagingdevice that captures in a direction directly behind of the mobileobject, in a direction right-diagonally behind of the mobile object, andin a direction left-diagonally behind of the mobile object, a thirdimaging device that captures in a direction directly right of the mobileobject, in the direction right-diagonally ahead of the mobile object,and in the direction right-diagonally behind of the mobile object, and afourth imaging device that captures in a direction directly left of themobile object, in the direction left-diagonally ahead of the mobileobject, and in the direction left-diagonally behind of the mobileobject.
 7. The control device according to claim 1, wherein the mobileobject is a vehicle, and the moving scene is a traveling scene of thevehicle.
 8. The control device according to claim 7, wherein thepredetermined rule defines that, when the vehicle enters a T-junction orwhen the vehicle restarts after a temporary stop, the directionleft-diagonally ahead of the vehicle and the direction right-diagonallyahead of the vehicle are set as targets of the distortion reductionprocessing more frequently than the direction directly right of thevehicle, the direction directly left of the vehicle, the directionright-diagonally behind of the vehicle, the direction left-diagonallybehind of the vehicle, and the direction directly behind of the vehicle.9. The control device according to claim 7, wherein the predeterminedrule defines that, when the vehicle enters a T-junction or when thevehicle restarts after a temporary stop, two regions in a directiondiagonally ahead of the vehicle out of two images acquired from twoimaging devices are set as targets of the distortion reductionprocessing at a same timing.
 10. The control device according to claim7, wherein the predetermined rule defines that, when the vehicle travelson a narrow road, the direction left-diagonally ahead of the vehicle andthe direction right-diagonally ahead of the vehicle are set as targetsof the distortion reduction processing more frequently than thedirection directly right of the vehicle, the direction directly left ofthe vehicle, the direction right-diagonally behind of the vehicle, thedirection left-diagonally behind of the vehicle, and the directiondirectly behind of the vehicle.
 11. The control device according toclaim 7, wherein the predetermined rule defines that, when the vehicletravels on a narrow road, the direction right-diagonally ahead of thevehicle and the direction left-diagonally ahead of the vehicle are setas targets of the distortion reduction processing more frequently thanthe direction directly right of the vehicle, the direction directly leftof the vehicle, the direction right-diagonally behind of the vehicle,the direction left-diagonally behind of the vehicle, and the directiondirectly behind of the vehicle.
 12. The control device according toclaim 7, wherein the predetermined rule defines that, when the vehicletravels on a narrow road, as compared with a case where the vehicletravels on a path other than the narrow road, positions in a verticaldirection of regions in the direction directly right of the vehicle, inthe direction directly left of the vehicle, in the directionright-diagonally behind of the vehicle, in the direction left-diagonallybehind of the vehicle, and in the direction directly behind of thevehicle are on a lower side.
 13. The control device according to claim7, wherein the predetermined rule defines that, when the vehicle travelson a narrow road, as compared with a case where the vehicle travels on apath other than the narrow road, positions in a vertical direction ofregions in the direction directly ahead of the vehicle, in the directionright-diagonally ahead of the vehicle, and in the directionleft-diagonally ahead of the vehicle are on a lower side.
 14. Thecontrol device according to claim 7, wherein the predetermined ruledefines that, when the vehicle travels on a narrow road, positions in avertical direction of regions in the direction right-diagonally ahead ofthe vehicle and in the direction left-diagonally ahead of the vehicleare on a higher side than positions in the vertical direction of regionsin the direction directly right of the vehicle and in the directiondirectly left of the vehicle.
 15. The control device according to claim7, wherein the predetermined rule defines that, when the vehicle turnsat an intersection, a direction diagonally ahead of the vehicle and on asame side of a direction to which the vehicle turns is set as a targetof the distortion reduction processing more frequently than thedirection directly behind of the vehicle and a direction diagonallybehind of the vehicle and on an opposite side of the direction to whichthe vehicle turns.
 16. The control device according to claim 15, whereinthe predetermined rule defines that, when the vehicle turns to adirection intersecting with an opposite lane at an intersection, thedirection directly ahead of the vehicle is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle and the direction diagonally behind ofthe vehicle and on the opposite side of the direction to which thevehicle turns.
 17. The control device according to claim 15, wherein thepredetermined rule defines that, when the vehicle turns at anintersection in a direction not intersecting an opposite lane, adirection diagonally behind of the vehicle and on the same side of thedirection to which the vehicle turns is set as a target of thedistortion reduction processing more frequently than the directiondirectly behind of the vehicle and a direction diagonally behind of thevehicle and on the opposite side of the direction to which the vehicleturns.
 18. The control device according to claim 7, wherein thepredetermined rule defines that, when the vehicle moves backward, thedirection directly behind of the vehicle, the direction right-diagonallybehind of the vehicle, and the direction left-diagonally behind of thevehicle are set as targets of the distortion reduction processing morefrequently than the direction right-diagonally ahead of the vehicle andthe direction left-diagonally ahead of the vehicle.
 19. The controldevice according to claim 18, wherein the predetermined rule is that,when the vehicle moves backward, the direction directly behind of thevehicle is set as a target of the distortion reduction processing morefrequently than the direction right-diagonally behind of the vehicle andthe direction left-diagonally behind of the vehicle.
 20. The controldevice according to claim 7, wherein the predetermined rule definesthat, when the vehicle moves backward, as compared with a case where thevehicle moves forward, positions in a vertical direction of regions inthe direction directly behind of the vehicle are on a lower side. 21.The control device according to claim 7, wherein the predetermined ruledefines that, when the vehicle performs a lane change, the directionright-diagonally ahead of the vehicle and the direction left-diagonallyahead of the vehicle are set as targets of the distortion reductionprocessing more frequently than a direction directly left or right ofthe vehicle and on a same side of a moving direction of the vehicle anda direction diagonally behind of the vehicle and on the same side of themoving direction the vehicle, and the direction directly left or rightof the vehicle and on the same side of the moving direction of thevehicle and the direction diagonally behind of the vehicle and on thesame side of the moving direction the vehicle are set as targets of thedistortion reduction processing more frequently than a directiondirectly left or right of the vehicle and on an opposite side of themoving direction of the vehicle and a direction diagonally behind of thevehicle and on the opposite side of the moving direction the vehicle.22. The control device according to claim 7, wherein the predeterminedrule defines that, when the vehicle performs a lane change, positions ina vertical direction of regions in a direction diagonally ahead of thevehicle and on an opposite side of a moving direction of the vehicle ison a lower side than positions in the vertical direction of regions in adirection diagonally ahead of the vehicle and on a same side of themoving direction of the vehicle.
 23. The control device according toclaim 1, wherein each of the one or more imaging devices is an imagingdevice to which a fisheye lens is attached.
 24. The control deviceaccording to claim 1, wherein the mobile object further includes anotherimaging device that captures an image with less distortion than the oneor more imaging devices, the image acquisition unit is configured toacquire an image of the external environment of the mobile object fromthe another imaging device, and the recognition unit is configured torecognize the external environment of the mobile object further based onan image from the another imaging device.
 25. A vehicle comprising thecontrol device according to claim
 1. 26. A non-transitory storage mediumcomprising a program for causing a computer to function as each unit ofthe control device according to claim
 1. 27. A method for controlling amobile object including one or more imaging devices, the methodcomprising: acquiring an image of an external environment of the mobileobject from the one or more imaging devices; performing distortionreduction processing for reducing distortion of an image for each of oneor more regions included in an image acquired from the one or moreimaging devices; and recognizing the external environment of the mobileobject based on an image on which the distortion reduction processinghas been performed, the one or more regions to be a target of thedistortion reduction processing is determined in accordance with apredetermined rule according to a moving scene of the mobile object.